New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Jump to bottom

Updating existing tokenizer naming, adding new tokenizer with BOS EOS tokens #91

Merged

KPHippe merged 2 commits into develop from patch/new_tokenizer

Mar 27, 2023

Collaborator

KPHippe commented Mar 13, 2023

Adding a new tokenizer with BOS/EOS tokens, can be loaded in same way as before


          Updating tokenizer to reflect actual vocab count, adding updated toke…

d0d343b

…nizer with BOS EOS tokens

KPHippe requested a review from braceal

March 13, 2023 21:48

Collaborator

braceal commented Mar 13, 2023 •

edited

Loading

If we change the name of the tokenizer file, we may also need to update the GenSLM class data to point to the correct tokenizer files. https://github.com/ramanathanlab/genslm/blob/main/genslm/inference.py#L22


          Did not add all files to previous commit... oops

d773d47

KPHippe merged commit 703bae0 into develop

KPHippe deleted the patch/new_tokenizer branch

March 27, 2023 16:11

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet